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ABSTRACT 

To assist NASA and the Air Force in determining how and when to invest in virtual environment 
(VE) technology for maintenance training, we identified possible roles for VE technology in such 
training, assessed its cost-effectiveness relative to existing technologies, and formulated 
recommendations for a research agenda that would address instructional and system development 
issues involved in fielding a VE training system. In the first phase of the study, we surveyed VE 
developers to forecast capabilities, maturity, and estimated costs for VE component technologies. 
We then identified maintenance tasks and their training costs through interviews with 
maintenance technicians, instructors, and training developers. Ten candidate tasks were selected 
from two classes of maintenance tasks (seven aircraft maintenance and three space maintenance) 
using five criteria developed to identify types of tasks most likely to benefit from VE training. 
Three tasks were used as specific cases for cost-benefit analysis. In formulating research 
recommendations, we considered three aspects of feasibility: technological considerations, cost- 
effectiveness, and anticipated R&D efforts. In this paper, we describe the major findings in each 
of these areas and suggest research efforts that we believe will help achieve the goal of a cost- 
effective VE maintenance training system by the next decade. 

1 INTRODUCTION 

Virtual environment (VE) technology holds great promise for maintenance and other technical 
training applications. VE capabilities (e.g., stereoscopic 360 degree field of regard, natural 
interactivity, tactile feedback, 3-D sound) can create a feeling of presence [1] that many believe 
will enhance the learning experience in ways that other technologies cannot [2, 3, 4]. The ability 
to faithfully simulate task environment characteristics makes VE attractive for training tasks that 
are performed under unusual conditions (e.g., zero gravity) or that involve the risk of injury or 
damage to equipment As a computer-based technology, it can be used in locations where space 
is limited (e.g., shipboard, in space) and can be configured to deliver training on large numbers 
of tasks that would otherwise require a suite of hardware trainers. A VE simulation can also 
accommodate more than one person at a time, and, through networking, participants need not all 
be in the same physical location. As part of a study to help NASA and the Air Force determine 
how and when to invest in VE technology for maintenance training, we were asked to (1) 
establish the need for VE technology in maintenance training by identifying categories of tasks 
for which VE would offer effective training, (2) assess VE’s cost-effectiveness relative to 
existing technologies, and (3) formulate recommendations for a research agenda that would 
address instructional and system development issues involved in fielding a VE training system. 

2 VE TECHNOLOGY SURVEY 

In the first phase of the study, we surveyed VE technology researchers and manufacturers to 
identify current and emerging capabilities, forecast maturity, and estimate costs for VE 
component technologies. The survey covered more than 50 organizations from government, 
industry, and academia. The survey findings are described in detail in a companion paper [5], 
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Here we cover two major aspects of the survey: key characteristics that define a VE simulation 
and critical VE technology research areas. 

2.1 Characteristics of VE Simulation 

Most VE systems share a cluster of essential elements that globally define the virtual 
environment. We briefly describe these characteristics to minimize confusion as to what 
constituted a VE system for the purposes of the study. 

Immersion in the virtual environment . VE systems can provide the user with a sense of 
immersion, that is, of being within the display rather than viewing it from outside. Immersion 
makes VE simulations much more realistic than through-the-screen simulations. Making the 
sense of immersion compelling requires coordination of sensory inputs to the user, so that the 
sensory attributes of virtual objects seem to be attached to those objects. It also requires the use 
of wide-field-of-view images so that the user's peripheral vision, not just central vision, is 
stimulated. 

Interactivity with elements of the virtual environment. The user should not only be a witness to 
events transpiring within the virtual environment, but a participant as well. Users must be able to 
navigate the virtual space and manipulate virtual objects within that space. Manipulations should 
have specifiable consequences that may vary with the simulation. For example, an astronaut 
standing on the surface of a simulated planet who fires a vertical thruster must accelerate at a rate 
consistent with the gravitational attraction of that planet. Interactivity also extends to other 
participants of the VE. An instructor must be able to change the viewpoint and orientation of a 
trainee as the task requires and examine the simulation from the viewpoint of the trainee. 

Sensory displays. The term display is used in its broadest sense, referring to making an 
impression on the senses. Specific examples of sensory displays include video screens, arrays of 
factors in the fingertips of gloves, and speaker arrays that produce localized sound sources in the 
VE. Early examples of VE training simulators will probably include only auditory and visual 
displays; the more sophisticated systems to follow will incorporate haptic displays, as well. VE 
simulations that lack haptic displays may be able to convey inertial and tactile information 
through other sensory channels (e.g., auditory or visual feedback). 

Remotely sensed and synthesized sensory images. Information presented on the sensory displays 
of VE training systems is likely to include audio, video, and possibly haptic images. These 
images will be primarily synthesized from a computer database, but remotely sensed images may 
be incorporated into the simulation in some instances. Synthesized images may be generated 
from a variety of databases, including CAD diagrams, electrical and hydraulic schematics, and 
other electronic blueprints used in the design of the objects represented in the VE simulation. 

2.2 VE Technology Recommendations 

VE technologies were divided into nine major components: 

• Visual display systems (head-mounted and CRT-based) 

• Position/orientation trackers 

• 3-D audio interfaces 

• 3-D/6-D input devices 

• Gesture-recognition input devices 

• Haptic interfaces 

• Automatic speech recognition systems 

• Computer hardware 

• Software. 
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Most VE technologies are in early phases of development, and current VE system components 
have restricted capabilities that limit the fidelity of the simulation. These limitations create 
sensory distortions that make many tasks difficult to perform in a VE. They can disorient the 
participant and even lead to simulator sickness. These and related problems limit the use of 
current VE technologies as training devices. Industry R&D will quickly improve many aspects 
of VE technology; however, much of the work will be aimed at producing low-cost components 
that will sell in high volume. In many cases, these components will not fulfill Air Force or 
NASA requirements for effective training devices. Further R&D funding will be needed to 
produce the high-quality devices needed for Air Force and NASA maintenance training systems. 
Here we describe those areas for which additional R&D efforts will be most critical. 

2.2.1 Visual Display Systems 

The low spatial resolution of present VE visual displays is a major limitation in application 
development. Only a few of the simplest maintenance training applications can be realized with 
present VE visual display resolution. Additional funding may be required to produce the small- 
diagonal, high-resolution head-mounted displays (HMDs) needed for maintenance training 
applications. Specifications of importance for VE visual display devices include field-of-view, 
spatial resolution, refresh rate, and color performance. Several technologies and visual display 
designs have promise for developing high-resolution VE visual displays. In our judgment, it is 
premature to choose a single VE visual display technology or design at this time. Consequently, 
our recommendations include research in: 

• High-resolution, small-diagonal HMD screens 

• Multiple- screen HMD configurations 

• Optical fibers for high-resolution, projection HMDs 

• Eye-tracking systems to be used in variable-resolution HMDs 

• Adjustable optics to provide wide-angle or high-resolution viewing. 

2.2.2 Position/Orientation Trackers 

Tracking the position and orientation of the VE-system participant is essential for developing 
highly interactive simulations. In most cases, six degrees of freedom must be tracked: three 
spatial-position coordinates and three orientation angles. Several technologies are currently 
being used for 6-D tracking in VE systems, including magnetic, ultrasound, mechanical, optical, 
and analog tracking devices. At present, each of these tracking technologies is under intense 
development. Major specifications to consider with tracking technologies are system range, 
resolution, repeatability, update rate, lag, and environmental robustness. The improvements 
needed in position/orientation-tracking systems include increases in tracking range, reductions in 
time delays, and minimization of encumbrances such as cabling. Efforts should be directed 
toward developing low-cost optical tracking systems and hybrid tracking systems using low- 
delay analog devices in conjunction with a remote- sensing system for periodic recalibration. 

2.2.3 Haptic Interfaces 

VE interfaces that involve the sense of touch are referred to as haptic interfaces. These interfaces 
fall into two main categories: force-feedback interfaces, which provide information about the 
mass and inertia of objects and forces applied to them, and tactile interfaces, which provide 
information about the shape and surface roughness of objects. Development of tactile-feedback 
interfaces is proceeding rapidly, with devices being fitted into gloves and other garments. The 
main problem facing the inclusion of these devices in VE simulations is to determine their proper 
use. Force-feedback interfaces are at a more fundamental stage of development, with 
applications being limited to providing force information to the hand and arm. Considerable 
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technological development will be required for them to become useful in VE systems. Task- 
specific force-feedback devices may be useful in many maintenance training applications. 

2.2.4 Computer Software 

The development of efficient and effective VE software will have the greatest impact on creating 
cost-effective VE maintenance training systems. Although VE software is being developed by 
many organizations, these packages do not fulfill the special needs of maintenance training 
applications. At present, multiple software packages are required to produce a VE application. 
First, virtual objects are created using graphics modeling software. Then, the simulation 
dynamics are programmed using another software package. The latter package usually controls 
the simulation as well, although additional software may be needed to provide image rendering. 
Ideally, VE software should provide each of these functions, as well as others, including: 

• Importation of CAD models and databases 

• Anthropometric modeling 

• Authoring environment 

• Networking capabilities. 

Although software is being developed for providing each of these capabilities, no fully integrated 
package has been implemented. Many enhancements will need to be made in VE software 
before cost-effective maintenance training simulations can be developed. 

3 MAINTENANCE TRAINING INTERVIEWS 

In a series of interviews, we collected information from experienced maintenance technicians, 
instructors, and instructional developers from the Air Force, NASA, and DoD contractors. The 
respondents nominated 19 aircraft maintenance task categories and 3 space maintenance task 
categories as possible candidates for VE training. Each task category was ranked on five 
selection criteria designed to identify those tasks that would benefit most from VE training. 

Table 1 shows the 10 highest-rated task categories. The interviews also yielded a wealth of 
information on maintenance task characteristics, current maintenance training practices and 
costs, training system descriptions, and trends that could affect future training. 


Table 1. Candidate Task Categories 

Space Maintenance: 

EVA 

Teleoperations 

Telerobotics 

Aircraft Maintenance: 

Engine Maintenance. 

Engine Run Test 
Fireguard 

Fuel System Maintenance 
Inspections 
Safety Procedures 
Marshaling 


To the extent possible, we also sought to identify particular capabilities and system performance 
levels that would be required to field a successful VE maintenance training system. The 
information gained from the interviews, together with our own knowledge of current VE 
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applications and maintenance training principles and systems, formed the basis of a cost- 
effectiveness analysis using three of die task categories identified. Rather than attempt a 
comprehensive discussion of our findings, we will report only our general conclusions. The 
interested reader is encouraged to read the full report (in preparation) for more detail. 

3.1 Maintenance Training Issues 

Three major conclusions were drawn from the interview data: 

A need exists for the kind of training VE offers. Safety and training impact are major factors in 
VE’s appeal. It could, for example, simulate the consequences of following improper procedures 
(e.g., shortcuts), give instructors more flexibility in monitoring and testing students’ 
performance, and provide students more meaningful feedback. 

VE could fill a gap between the two predominant current training technologies. VE combines 
many of die benefits of hardware-based simulation with those of computer-based delivery, 
providing higher-fidelity simulation than interactive videodisc (TVD) lessons and more flexibility 
and instructional features than hardware-based simulators. 

VE-based training could be cost-effective for many applications. Using the cost data collected in 
the interviews, we conducted a cost-effectiveness analysis comparing VE with current training 
technologies. In each of three sample applications, the results suggest that VE-based training 
could be a cost-effective addition to, or replacement for, existing training systems. [The reader is 
encouraged to refer to the project final report (in preparation) for details and cost assumptions.] 

3.1.1 Aircraft Maintenance 

Technical schools and field training detachments (FTDs) are hampered by an inadequate supply 
of up-to-date equipment. OJT suffers from a lack of standardization and instructional feedback. 
Training development and upgrading are costly processes that often lag behind training needs. 
Training systems and courseware have difficulty keeping pace with weapons system 
modifications. High-fidelity hardware/software simulators, although ideal for small numbers of 
students, are extremely costly and not available in large enough quantities to accommodate the 
large numbers of maintenance students. Low-end systems do not provide the fidelity to train to 
masteiy on many tasks. Our look at future trends in aircraft maintenance identified several 
potential challenges, including discontinuation of the use of actual aircraft for training, reduction 
of equipment time available for OJT, and further consolidation of maintenance specialties. 

These problems are sufficiently acute to warr ant look ing into new technologies such as VE to 
ease the training burden. 

VE systems will not be inexpensive; currently, a system containing the appropriate capabilities 
(if available) would be far too costly to compete with other technologies. Our data suggest, 
however, that cost should not be a deterrent to exploring VE as a future alternative. Our 
estimates— using costs projections for VE technologies at maturity— show that VE could be used 
cost-effectively throughout much of training. VE system development, maintenance, and 
upgrade costs are expected to be well within the range of costs currently expended for IVD 
courseware and hardware/software trainers. We also expect VE to enjoy life-cycle cost savings 
and benefits comparable to those attributed to other computer-based training technologies [6] [7]. 

Whether VE simulation will make a cost-effective training delivery tool depends on several 
factors, including the required capabilities of the system, the nature of the tasks to be trained, and 
the alternative means of delivering the instruction. The “ideal” solution might employ several 
levels of systems. The concept of using multiple levels of simulation is being employed in 
aircrew training to eliminate training bottlenecks on full-fidelity flight simulators. The idea is to 
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use lower-fidelity “part-task” simulators to teach cockpit familiarization and basic procedures 
before moving on to the high-fidelity simulator. In maintenance training, the role of part-task 
trainer could be filled by VE systems. 

3.1.2 Space Maintenance 

NASA has a continuing interest in the development of VE simulations for use in both its ground 
operations and space missions. Much of NASA’s interest stems from the fact that it is difficult 
and expensive to practice on earth procedures meant to be performed in a zero-g environment. 
Extravehicular activity (EVA) tasks, for example, were forecast to be prime candidates for VE 
training. Currently, training is conducted for specific missions on full-scale replicas and in 
neutral-buoyancy simulators. As discovered on a recent satellite rescue mission, the practice 
provided on these systems may not be sufficient to perform tasks such as satellite coupling in 
space. It is hoped that VE simulations will provide an appropriate representation of the physics 
of a zero-gravity environment, thereby supporting mission planning and rehearsal, as well as 
general training. Another task (planned for the space station) is the use of extravehicular robots 
controlled by technicians inside the space station. Although no training for this task category is 
currently conducted, NASA is experimenting with VE and dome projection systems to determine 
which provides a more accurate representation of the task environment. 

The small student population and the limited number, special purpose, and short duration of 
space missions have enabled space maintenance training to get along with a small number of 
very expensive trainers. This situation is likely to change with the construction and habitation of 
the space station. New classes of maintenance tasks will have to be taught to larger numbers of 
students. It is unclear whether current training practices (e.g., zero-g profile flights) or 
simulators (e.g., neutral-buoyancy or dome simulators) can handle the increased training needs. 
Such systems are expensive, and they have inherent limitations in simulating space maintenance 
tasks. VE simulations (albeit with their own limitations) may prove to be a relatively 
inexpensive alternative to hardware-based training systems. Because the physical laws that 
govern interactions among objects in a virtual environment are part of the program that controls 
the simulation, VE can simulate interactions in zero-g (or other gravitational) environments. 

With improvements in force-feedback systems, VE systems will also be able to simulate inertial 
characteristics of objects, something that is difficult to achieve with conventional simulations. 

Another factor that will become increasingly important as space missions get longer is the ability 
to maintain skills acquired before the mission. On long missions, skill levels developed during 
the ground-based rehearsal phase can diminish unless some means is provided to maintain the 
skill. Mass and space constraints eliminate hardware-intensive simulations for skill maintenance 
during space flight. An alternative would be a general-purpose VE simulator. By deploying the 
appropriate task simulation in the onboard VE system, an astronaut would be able to maintain the 
skill level achieved on earth. Moreover, in an anomalous situation, a ground station could 
transmit data for generating a new scenario, which could then be used to guide the astronaut 
through practice runs on emergency procedures that had not been rehearsed on earth. 

3.2 Training Research Recommendations 

To provide effective VE maintenance training systems, research will be needed in several areas, 
including user interaction methods, learning benefits of VE, instructional strategies, and testing. 
Recommendations were made in each of these areas. 

3.2.1 User Interaction Methods 

Effective use of VE devices (e.g., 3-D/6-D input devices) and techniques (e.g., virtual menu 
screens, voice commands, and hand gestures) will require an understanding of how simulator 
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interactions can best be performed. Most interactions can be performed by a variety of methods. 
Present VE system interactions are generally restricted by the available user interface devices 
and styles. A number of conventions have been created for these interactions, but it is generally 
accepted that these conventions do not produce optimal interaction. Information from 
assessment of VE system interactions can be used to greatly enhance usability by both 
experienced and naive participants. Studies should be conducted to develop and evaluate: 

• Methods for navigating within a simulated environment 

• Methods for manipulating virtual objects 

• Command modes for steering simulations 

• Methods for interaction within multi-participant applications 

3.2.2 Effects of VE Systems on Learning 

VE systems have the potential to enhance many aspects of learning (e.g., complex knowledge 
structures, procedural and spatial skills [8], pattern recognition) as compared with other modes of 
instruction. Unfortunately, the effects of VE simulation on the acquisition of knowledge and 
perceptual, cognitive, and motor skills are not well understood. Studies should be conducted to 
understand the role(s) that VE systems can play in achieving various training objectives, and the 
advantages of VEs over other technologies for the development and retention of knowledge and 
skills and their transfer to the actual environment. Factors that may affect learning include: 

• Immersion (HMD) versus window-on-the-world (CRT) 

• 2-D versus 3-D display 

• Dynamic versus static objects 

• Interactive versus passive participation 

• Effects of scale and perspective 

• Effects of varying fidelity. 

The advantage of VE technology over other training media will depend on the kind of task being 
learned and the stage of skill acquisition. For example, both electronics troubleshooters and jet 
engine mechanics must have a sophisticated functional representation, or mental model, of the 
target system. A logical case could be made that students in either or both disciplines would 
develop a more complete or useful mental model of the target system from a VE simulation than 
from the same information presented via another medium (e.g., 2-D model on a CRT). Empirical 
data on this issue are lacking, however. It is quite possible that the mechanical task environment 
requires a spatial component in the mechanic’s mental model that is not present (or necessary) in 
the troubleshooter’s model. It remains to be demonstrated that VE training in a 3-D world would 
facilitate the acquisition and use of that representation on the job. The availability of a third 
dimension can serve to simplify the presented mental model in cases where it is able to reveal 
patterns or relationships that are hard to discern in two-dimensional representations (e.g., 
molecular structures). If, however, representation of a third dimension leads training developers 
to implement more complex mental models, this virtue could be lost. Research investigating 
whether, and under what conditions, a VE simulation is more effective than CRT-based 
simulations in conveying a mental model would determine an important role for VE training 
systems. 

Another issue concerns possible advantages of acquiring additional mental representations of the 
content to be learned. A mechanic’s expertise is tied closely to perceptual skills (e.g., hearing a 
“faulty” sound, feeling a warp, seeing a crack, or estimating distance). If the sense of presence 
and kinesthetic experience with the virtual system facilitate learning of these skills, then a VE 
simulation would have advantages over a conventional CRT display of the same 3-D graphics. 

On the other hand, if the value of VE technology for promoting learning about systems lies in the 
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provision of interactive three-dimensional graphics, the use of VE technologies (e.g., head- 
mounted display, 3-D audio) that are more costly is unnecessary. 

3.2.3 Instructional Strategies 

In our report, we assume that a typical VE training system will incorporate several advanced 
technologies (e.g., intelligent authoring and delivery, speech recognition, software simulation, 
and modeling technologies). Although some of these components have already been synthesized 
into prototype training systems, fielding a VE system will not be simply a technical matter. VE 
will add a level of complexity to training delivery that is not well understood. An important 
implementation issue will be understanding how best to employ VE technology to achieve a 
given training objective. Experience with some nontraining VE applications suggests that users 
should be free to explore the virtual world without encumbrance. Training studies using other 
technologies, however, indicate that guided exploration and structured tutorial are more effective 
for some objectives. VE development efforts have not yet dealt with the problems associated 
with such issues. Questions regarding how, and how much, the system should intrude into the 
environment will be important in determining both the effectiveness and acceptability of VE as a 
training tool. Studies should be conducted to develop effective training methods in a VE. 

A related issue involves the stage(s) of training (or skill development) in which VE is most 
effective. Our interview respondents suggested that for some tasks, VE can be used early in 
training to familiarize the student with the job environment. For other tasks, VE would be most 
effective in hands-on training to develop procedural, perceptual, and cognitive skills. Research is 
needed to identify task characteristics that determine the appropriate timing and quantity of VE 
training. Studies should be conducted to develop guidelines specifying how VE and other 
technologies can be used in concert to optimize the benefits of each. 

3.2.4 Testing Studies 

Testing is an area in which VE shows much promise. Current commonly used methods tend to 
suffer from either a lack of content validity (e.g., use of written tests when task skills are at issue) 
or a lack of standardization (e.g., lack of reliable scoring for performance tests). VE could be 
used to test the qualifications of a student for promotion to the next stage of training, and to 
assess the continuing competency of journeyman technicians. VE might also offer a more 
effective alternative to current methods in the administration of aptitude and job- screening tests. 
Although not the focus of this study, the trend toward performance-based aptitude and screening 
tests clearly suggests a role for the kind of simulation offered by VE systems. 

4 CONCLUSIONS AND NEXT STEPS 

Although its role is clear in training that otherwise would not be feasible, the utility and cost- 
effectiveness of VE as a general-purpose maintenance training tool remains untested. Several 
obstacles must be overcome before VE can offer the benefits envisioned by its promoters. To be 
useful as a maintenance training delivery system, VE systems must achieve a higher level of 
technical sophistication than is currently available (e.g., higher visual resolution), and they must 
be cost-effective in comparison with alternative training delivery systems. Moreover, VE 
systems will have to prove their effectiveness for learning. This will mean developing a research 
base from which we can draw inferences about which VE and companion technologies are 
appropriate for a given application, and developing guidelines for effective feedback and user 
interaction protocols (e.g., how should a user move around: gestures, voice commands, physical 
movement?). It will also require an understanding of how this new form of simulation affects 
knowledge (e.g., mental model) and skill (e.g., spatial reasoning) acquisition. 
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Virtual environment systems are expected to become commonplace within the next decade, so it 
is essential that government and industry prepare to exploit this technology as it matures. One 
major question that must be answered concerns whether VE systems will provide more efficient 
and effective means of accomplishing specific training goals than comparable traditional 
systems. Like most questions of this complexity, this one has no simple answer, VE systems 
will be cost-effective in some applications, but not in others. We are currently formulating a plan 
for the suggested VE research, focusing on the development of demonstration systems for 
selected maintenance training applications. The plan will include recommendations for research 
priorities and sequencing, as well as the coordination of efforts among DoD and NASA 
organizations. TTie plan addresses hardware and software procurement and facilities 
requirements, including the relative advantages of centralized versus decentralized facilities. It 
also considers the impact of ongoing VE R&D as well as training R&D in closely related areas. 
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